Databricks Integration
Airflow Schedules, Databricks Computes
The same orchestrator/processor split covered for EMR and Glue applies here: Databricks is a managed Spark platform with its own job scheduler, notebooks, and clusters. Airflow's role is the same as always - decide when a Databricks job runs, retry it if it fails, and fit it into a bigger pipeline alongside everything else.
Same honest treatment as EMR/Redshift/Glue: a real Databricks workspace and access token would be needed to run this for real, neither available in this sandbox. Code-only.
Running an Existing Job
The most common pattern — a job is already defined in the Databricks UI, and Airflow just triggers a run of it:
from airflow.providers.databricks.operators.databricks import DatabricksRunNowOperator
run_databricks_job = DatabricksRunNowOperator(
task_id="run_daily_transform_job",
databricks_conn_id="databricks_default",
job_id=123456789,
notebook_params={"run_date": "{{ ds }}"},
)
Submitting a One-Off Run
For a job that doesn't have a persistent job definition in Databricks — Airflow specifies the full cluster and notebook/script inline:
from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator
submit_transform_run = DatabricksSubmitRunOperator(
task_id="submit_transform_job",
databricks_conn_id="databricks_default",
new_cluster={
"spark_version": "15.4.x-scala2.12",
"node_type_id": "i3.xlarge",
"num_workers": 2,
},
notebook_task={
"notebook_path": "/Repos/data-eng/transform_sales",
"base_parameters": {"run_date": "{{ ds }}"},
},
)
DatabricksSubmitRunOperator provisions a fresh cluster for this run only (job clusters, billed only while running) — the same "don't pay for idle compute" idea as EMR's transient clusters, just Databricks-managed instead of raw EC2.
Databricks Workflows Orchestrating Airflow-Adjacent Steps
Newer Databricks deployments increasingly use Databricks Workflows (Databricks' own native orchestrator) for pipelines that live entirely within Databricks — multiple notebook tasks with dependencies, similar in spirit to a small Airflow DAG. When that's the case, Airflow's job usually simplifies to one task: trigger the whole Databricks Workflow and wait for it, rather than modeling each notebook as a separate Airflow task.
from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup
with DatabricksWorkflowTaskGroup(
group_id="databricks_workflow",
databricks_conn_id="databricks_default",
job_clusters=[...],
) as workflow:
... # individual notebook tasks, mirroring the Databricks Workflow's own task graph
Prefer
DatabricksRunNowOperator (an existing job) whenever the Databricks side is stable and versioned in the Databricks UI/Git - it keeps cluster configuration and notebook logic owned by the team that maintains the Databricks side. Reach for DatabricksSubmitRunOperator only for genuinely ad hoc, one-off runs that don't warrant a persistent job definition.